Distributed Multi-Robot Learning using Particle Swarm Optimization

This thesis studies the automatic design and optimization of high-performing robust controllers for mobile robots using exclusively on-board resources. Due to the often large parameter space and noisy performance metrics, this constitutes an expensive optimization problem. Population-based learning techniques have been proven to be effective in dealing with noise and are thus promising tools to approach this problem. We focus this research on the Particle Swarm Optimization (PSO) algorithm, which, in addition to dealing with noise, allows a distributed implementation, speeding up the optimization process and adding robustness to failure of individual agents. In this thesis, we systematically analyze the different variables that affect the learning process for a multi-robot obstacle avoidance benchmark. These variables include algorithmic parameters, controller architecture, and learning and testing environments. The analysis is performed on experimental setups of increasing evaluation time and complexity: numerical benchmark functions, high-fidelity simulations, and experiments with real robots. Based on this analysis, we apply the distributed PSO framework to learn a more complex, collaborative task: flocking. This attempt to learn a collaborative task in a distributed manner on a large parameter space is, to our knowledge, the first of such kind. In addition, we address the problem of noisy performance evaluations encountered in these robotic tasks and present a %new distributed PSO algorithm for dealing with noise suitable for resource-constrained mobile robots due to its low requirements in terms of memory and limited local communication

Distributed Particle Swarm Optimization for limited-time adaptation with real robots

Evaluative techniques offer a tremendous potential for online controller design. However, when the optimization space is large and the performance metric is noisy, the overall adaptation process becomes extremely time consuming. Distributing the adaptation process reduces the required time and increases robustness to failure of individual agents. In this paper, we analyze the role of the four algorithmic parameters that determine the total evaluation time in a distributed implementation of a Particle Swarm Optimization (PSO) algorithm. For an obstacle avoidance case study using up to eight robots, we explore in simulation the lower boundaries of these parameters and propose a set of empirical guidelines for choosing their values. We then apply these guidelines to a real robot implementation and show that it is feasible to optimize 24 control parameters per robot within 2 h, a limited amount of time determined by the robots' battery life. We also show that a hybrid simulate-and-transfer approach coupled with a noise-resistant PSO algorithm can be used to further reduce experimental time as compared to a pure real-robot implementatio

Distributed Particle Swarm Optimization for limited-time adaptation with real robots

Evaluative techniques offer a tremendous potential for online controller design. However, when the optimization space is large and the performance metric is noisy, the overall adaptation process becomes extremely time consuming. Distributing the adaptation process reduces the required time and increases robustness to failure of individual agents. In this paper, we analyze the role of the four algorithmic parameters that determine the total evaluation time in a distributed implementation of a Particle Swarm Optimization (PSO) algorithm. For an obstacle avoidance case study using up to eight robots, we explore in simulation the lower boundaries of these parameters and propose a set of empirical guidelines for choosing their values. We then apply these guidelines to a real robot implementation and show that it is feasible to optimize 24 control parameters per robot within 2 h, a limited amount of time determined by the robots' battery life. We also show that a hybrid simulate-and-transfer approach coupled with a noise-resistant PSO algorithm can be used to further reduce experimental time as compared to a pure real-robot implementation

The Role of Environmental and Controller Complexity in the Distributed Optimization of Multi-Robot Obstacle Avoidance

The ability to move in complex environments is a fundamental requirement for robots to be a part of our daily lives. Increasing the controller complexity may be a desirable choice in order to obtain an improved performance. However, these two aspects may pose a considerable challenge on the optimization of robotic controllers. In this paper, we study the trade-offs between the complexity of reactive controllers and the complexity of the environment in the optimization of multi-robot obstacle avoidance for resource-constrained platforms. The optimization is carried out in simulation using a distributed, noise-resistant implementation of Particle Swarm Optimization, and the resulting controllers are evaluated both in simulation and with real robots. We show that in a simple environment, linear controllers with only two parameters perform similarly to more complex non-linear controllers with up to twenty parameters, even though the latter ones require more evaluation time to be learned. In a more complicated environment, we show that there is an increase in performance when the controllers can differentiate between front and backwards sensors, but increasing further the number of sensors and adding non-linear activation functions provide no further benefit. In both environments, augmenting reactive control laws with simple memory capabilities causes the highest increase in performance. We also show that in the complex environment the performance measurements are noisier, the optimal parameter region is smaller, and more iterations are required for the optimization process to converge

A Comparison of PSO and Reinforcement Learning for Multi-Robot Obstacle Avoidance

The design of high-performing robotic controllers constitutes an example of expensive optimization in uncertain environments due to the often large parameter space and noisy performance metrics. There are several evaluative techniques that can be employed for on-line controller design. Adequate benchmarks help in the choice of the right algorithm in terms of final performance and evaluation time. In this paper, we use multi-robot obstacle avoidance as a benchmark to compare two different evaluative learning techniques: Particle Swarm Optimization and Q-learning. For Q-learning, we implement two different approaches: one with discrete states and discrete actions, and another one with discrete actions but a continuous state space. We show that continuous PSO has the highest fitness overall, and Q-learning with continuous states performs significantly better than Q-learning with discrete states. We also show that in the single robot case, PSO and Q-learning with discrete states require a similar amount of total learning time to converge, while the time required with Q-learning with continuous states is significantly larger. In the multi-robot case, both Q-learning approaches require a similar amount of time as in the single robot case, but the time required by PSO can be significantly reduced due to the distributed nature of the algorithm

Distributed vs. Centralized Particle Swarm Optimization for Learning Flocking Behaviors

In this paper we address the automatic synthesis of controllers for the coordinated movement of multiple mobile robots. We use a noise-resistant version of Particle Swarm Optimization to learn in simulation a set of 50 weights of a plastic artificial neural network. Two learning strategies are applied: homogeneous centralized learning, in which every robot runs the same controller and the performance is evaluated externally with a global metric, and heterogeneous distributed learning, in which robots run different controllers and the performance is evaluated independently on each robot with a local metric. The two sets of metrics enforce Reynolds’ flocking rules, resulting in a good correspondence between the metrics and the flocking behaviors obtained. Results demonstrate that it is possible to learn the collective task using both learning approaches. The solutions from the centralized learning have higher fitness and lower standard deviation than those learned in a distributed manner. We test the learned controllers in real robot experiments and also show in simulation the performance of the controllers with increasing number of robots

Noise-Resistant Particle Swarm Optimization for the Learning of Robust Obstacle Avoidance Controllers using a Depth Camera

The Ranger robot was designed to interact with children in order to motivate them to tidy up their room. Its mechanical configuration, together with the limited field of view of its depth camera, make the learning of obstacle avoidance behaviors a hard problem. In this article we introduce two new Particle Swarm Optimization (PSO) algorithms designed to address this noisy, high-dimensional optimization problem. Their aim is to increase the robustness of the generated robotic controllers, as compared to previous PSO algorithms. We show that we can successfully apply this set of PSO algorithms to learn 166 parameters of a robotic controller for the obstacle avoidance task. We also study the impact that an increased evaluation budget has on the robustness and average performance of the optimized controllers. Finally, we validate the control solutions learned in simulation by testing the most robust controller in three different real arenas

Distributed Particle Swarm Optimization - Particle Allocation and Neighborhood Topologies for the Learning of Cooperative Robotic Behaviors

In this article we address the automatic synthesis of controllers for the coordinated movement of multiple mobile robots, as a canonical example of cooperative robotic behavior. We use five distributed noise-resistant variations of Particle Swarm Optimization (PSO) to learn in simulation a set of 50 weights of an artificial neural network. They differ on the way the particles are allocated and evaluated on the robots, and on how the PSO neighborhood is implemented. In addition, we use a centralized approach that allows for benchmarking with the distributed versions. Regardless of the learning approach, each robot measures locally and individually the performance of the group using exclusively on-board resources. Results show that four of the distributed variations obtain similar fitnesses as the centralized version, and are always able to learn. The other distributed variation fails to properly learn on some of the runs, and results in lower fitness when it succeeds. We test systematically the controllers learned in simulation in real robot experiments

Analysis of Fitness Noise in Particle Swarm Optimization: From Robotic Learning to Benchmark Functions

Population-based learning techniques have been proven to be effective in dealing with noise and are thus promising tools for the optimization of robotic controllers, which have inherently noisy performance evaluations. This article discusses how the results and guidelines derived from tests on benchmark functions can be extended to the fitness distributions encountered in robotic learning. We show that the large-amplitude noise found in robotic evaluations is disruptive to the initial phases of the learning process of PSO. Under these conditions, neither increasing the population size nor increasing the number of iterations are efficient strategies to improve the performance of the learning. We also show that PSO is more sensitive to good spurious evaluations of bad solutions than bad evaluations of good solutions, i.e., there is a non-symmetric effect of noise on the performance of the learning

Distributed Particle Swarm Optimization using Optimal Computing Budget Allocation for Multi-Robot Learning

Particle Swarm Optimization (PSO) is a population-based metaheuristic that can be applied to optimize controllers for multiple robots using only local information. In order to cope with noise in the robotic performance evaluations, different re-evaluation strategies were proposed in the past. In this article, we apply a statistical technique called Optimal Computing Budget Allocation to improve the performance of distributed PSO in the presence of noise. In particular, we compare a distributed PSO OCBA algorithm suitable for resource-constrained mobile robots with a centralized version that uses global information for the allocation. We show that the distributed PSO OCBA outperforms a previous distributed noise-resistant PSO variant, and that the performance of the distributed PSO OCBA approaches that of the centralized one as the communication radius is increased. We also explore different parametrizations of the PSO OCBA algorithm, and show that the choice of parameter values differs from previous guidelines proposed for stand-alone OCBA